Enabling Checkpoint Caches
You can create a checkpoint cache to save copies of the model at different epochs. Saved checkpoints are marked in bold on the plotted graph during training and can also be reviewed in the Training dialog for any training session. Do the following to open the Training dialog, which is shown below:
- In the Deep Learning Tool, right-click a model in the Model list on the Model Overview panel and then choose Display Training History and the required the training session in the drop-down menu.
- In the Segmentation Wizard, right-click a model on the Models tab and then choose Display Training History and the required the training session in the drop-down menu.
Training dialog
How to Enable Checkpoint Caches
- Go to Preferences > Deep Learning to open the Checkpoint cache preferences, shown below.
- Check the Enable checkpoint cache option.
- Set a per model limit for the maximum cache disk space in KB, MB, GB, or TB.
You should select a per model disk space as per the type of model you usually train and the number of checkpoints you want to save. For example, each copy of a U-Net model with a depth level of 3 and an initial filter count of 64 takes approximately 15 MB, while each copy of a U-Net model with an input dimension of 3 slices, a depth level of 5, and an initial filter count of 64 takes approximately 250 MB.
Note Models will be saved at a checkpoint each time that it has a better score until the cache limit is reached. Afterwards, the model with the lowest score will be cleared so that the new checkpoint can be saved.
- Click to OK to save your changes.
